Types of Indexes
Types of Merges
Indexing-Related Performance Counters
Failure Handling
Property Cache
Storing the words and properties extracted by the CiDaemon process in indexes is referred to as indexing. An index is a special data structure that is used to satisfy queries efficiently.
As documents in the corpus (Web site) are modified, the indexing program is notified of the updates and those documents enter a change queue. The CiDaemon process retrieves documents from the change queue in “first-in-first-out” order and filters them. The resulting words and properties are then added to the index. If there are many documents waiting in the change queue, there will be a delay before the index has up-to-date information about those documents.
Types of IndexesThere are three types of indexes: word lists, shadow indexes, and a master index. Words and properties extracted from a document first appear in a word list, then move to a shadow index, and finally move to the master index. This organization is optimized for query responsiveness and performance. It also ensures optimal resource usage. Even though there are multiple indexes internally, these details are completely hidden from the user. The user sees only a list of documents that satisfy the query that was posted.
Word lists are small, in-memory indexes. Each word list contains data for a small number of documents. As soon as a document is filtered, its data is stored in a word list. Creation of a word list is very quick and does not require updating any on-disk data. It is used as a temporary staging area during indexing.
There are several registry parameters that control word list behavior. All the keys are under the registry path \HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Content Index. The following table shows the registry parameters and explanations.
| Parameter | Explanation |
|---|---|
| MaxWordLists | Maximum number of word lists. If the number exceeds this, a shadow merge will be performed. |
| MaxWordlistSize | Maximum recommended size of a single word list. If the size of a word list exceeds this value, a new one will be created. This is an internal value and must not be changed. |
| MinSizeMergeWordlists | If the combined size of all word lists exceeds this number, a shadow merge will be performed. |
Once the number of word lists exceeds the MaxWordLists parameter, the word lists are merged into a shadow index. This merge process is called the shadow merge. Although the data in word lists is compressed to some extent, the compression is not very high because word lists are temporary structures. Because word lists are in-memory strucures, documents in a word list must be refiltered whenever IIS is restarted. The refiltering is automatically detected and performed by the Microsoft Index Server engine.
When data for an index is stored on disk, it is called a persistent index. Unlike word lists, which are in-memory indexes, a persistent index survives shutdowns and restarts. Persistent-index data is stored in a highly compressed format. There are two types of persistent indexes:
A shadow index is a persistent index created by merging word lists and sometimes other shadow indexes into a single index. There can be multiple shadow indexes in the catalog.
A master index is a persistent index that contains the indexed data for a large number of documents. This is usually the largest persistent data structure. In an ideal state, this is the only index present, because all the indexed data is stored in the master index and there are no shadow indexes or word lists. The data is highly compressed.
A master index is created by master merge, which merges all the shadow indexes and the current master index (if any) into a new master index. After the master merge, all the source indexes are deleted and only the new master index will be left. In this state, queries are resolved most efficiently.
The total number of persistent indexes (shadow indexes and master index) in a catalog cannot exceed 255.
Types of MergesThe process of combining data from multiple indexes into a single index is called merging. Merging results in getting rid of some redundant data and also freeing up resources. Queries are also resolved faster with fewer indexes. There are three types of merges:
After a merge completes, the multiple source indexes are replaced by a single target index.
Combining multiple word lists and shadow indexes into a single shadow index is called a shadow merge. A shadow merge is performed to free up memory used by word lists and also to make the filtered data persistent; it is usually a quick operation.
In the most common case, the source indexes for a shadow merge are word lists. However, if the total number of shadow indexes exceeds MaxIndexes, some of the shadow indexes are also used as source indexes. Shadow indexes are also used as source indexes during an annealing merge.
A shadow merge is triggered by one of the following conditions:
For a master merge, the source indexes are all of the existing shadow indexes and the current master index (if any). At the end of a master merge, all the source indexes are replaced by a single target master index. Although the master merge itself is a very resource-intensive (both for CPU and disk space) operation, after the completion of a master merge, resources are freed up. A lot of the redundant data is deleted and queries run faster.
Depending upon the size of the source indexes, a master merge can be a very long-running operation. However, it is fully restartable after failures and shutdowns. A master merge will continue from where it left off.
Whenever a master merge is started, restarted, or paused, an event is written to the event log. There are several reasons for starting a master merge. Some reasons for starting a master merge follow.
An annealing merge is a special kind of shadow merge performed when the system is idle for a cerain length of time and the total number of persistent indexes exceed MaxIdealIndexes. The registry parameter MinMergeIdleTime specifies the percentage of CPU time that must be idle during a time period to trigger an annealing merge. An annealing merge improves query performance and disk space usage by reducing the number of shadow indexes.
Indexing-Related Performance CountersThe following performance counters are related to indexing and merging. They are all present under the Content Index object.
| Counter Name | Explanation |
|---|---|
| Index Size | Total size of all the persistent indexes in megabytes. |
| Persistent Indexes | Total number of persistent indexes. |
| Merge Progress | Percentage of merge completed. |
| Word lists | Total number of word lists. |
Failure HandlingThere can be several kinds of failures during filtering, indexing, and merging. Except after a hardware failure, indexes are fully recoverable. Indexing and merging are temporarily paused if memory load is very high. Failed operations are retried later.
A merge is not started if the free disk space is very low on the catalog drive. However, it is possible that the drive may run out of free space while the merge is happening. A shadow merge will be aborted and retried after disk space is freed up. A master merge is not ended but temporarily paused. An event is written to the Windows NT event log that the Master Merge is paused. A disk full event will also be written. The administrator should free up disk space by moving or deleting data files from the corpus. Do not delete any files under the Index Server Catalog Directory. The system will automatically detect when enough disk space has been freed and restart the operations.
Property CacheThe property cache is an on-disk store optimized to speed up the retrieval of frequently retrieved values such as Path, Abstract, Title, Attributes, Last Write time stamp, File Size, and some values for internal use only. In a future release, administrators will be able to configure the property cache for storing custom properties.
The property cache is also a large data structure, its size being comparable to that of the master index. The registry parameter PropertyStoreMappedCache controls how much of the property cache is always kept in memory. On large index servers, setting this value higher will yield better performance. If the physical memory is not adequate, the performance might suffer.
During a dirty shutdown, the property cache may become corrupted. During startup after a dirty shutdown, a consistency check is performed on the property cache and if any problems are detected, they are fixed. However, some irrepairable inconsistencies may occur. In that case, all existing index data is thrown out and documents automatically re-index. Please see the Error Detection and Recovery page for more information.
Events are written to the event log when a recovery operation is performed on the property cache.
© 1996 by Microsoft Corporation. All rights reserved.